Skip doc-comment copying and presize token arrays when tokenizing - #23200
Skip doc-comment copying and presize token arrays when tokenizing#23200staabm wants to merge 1 commit into
Conversation
The scanner copied every doc comment into CG(doc_comment) even in non-parser mode (token_get_all/PhpToken::tokenize/highlighting), where nothing consumes it and it is immediately discarded; only do it when compiling. Also presize the tokenizer result array (one token per ~5 source bytes) and the dedup hash instead of growing them by doubling.
|
to be honest, I tried to benchmark this change and was not able to successfully get a meaningful result. |
| zend_hash_init(&interned_strings, 128, NULL, NULL, 0); | ||
| /* Rough estimate: one token per ~5 source bytes; presizing avoids | ||
| * repeated doubling of the result array. */ | ||
| array_init_size(return_value, ZSTR_LEN(source) / 5 + 8); |
There was a problem hiding this comment.
Are you sure about this? Now, I see the reason why this improves performance somehow, as claude claims, but this completely depends on the input.
For example, a huge inline HTML, or very long strings only produce a few tokens. but the first insert will allocate packed array capacity based on source bytes. This can turn valid large inputs into avoidable OOM cases that we don't want to see.
As far as I can tell, roughly >160 MiB already exceeds HT_MAX_SIZE on 32 bits.
I didn't run any benchmark tho, so someone need to prove the improvements exists :)
There was a problem hiding this comment.
I'm also skeptical about this... the ~5 figure really comes out of thin air.
Girgias
left a comment
Girgias
left a comment
There was a problem hiding this comment.
I'm frankly not interested in reviewing code generated by an LLM.
And if you can't even produce a benchmark showing in what cases this improves then what is even the point?
This might cause regressions elsewhere for all we know.
5 participants
disclaimer: this change was generated by claude opus. I have little experience with php-src development
The scanner copied every doc comment into CG(doc_comment) even in non-parser mode (token_get_all/PhpToken::tokenize/highlighting), where nothing consumes it and it is immediately discarded; only do it when compiling. Also presize the tokenizer result array (one token per ~5 source bytes) and the dedup hash instead of growing them by doubling.